Assessing the quality of evidence in evidence-based policy: why, how and when?

Interest in the issue of ‘research quality’ is at an all time high. Undoubtedly, one of the key spurs to the quest for higher standards in social research is the evidence-based policy movement. The chosen instrument for figuring out best-possible, future interventions in a particular policy domain is the systematic review of all first-rate, bygone evidence from previous studies in that realm. In trying to piece together the evidence that should carry weight in policy formation, a key step in the logic is to provide an ‘inclusion criterion’ as a means identifying those existing studies upon which most reliance should be placed. This paper examines some recent yardsticks used to sort the evidential sheep from the research goats by questioning why, how and when such research standards should be brought to bear. It concludes that the drive to cast standards as formal checklists of quality indicators is premature, and that appraising quality is not and cannot be a technical preliminary to research synthesis. Open and critical debate on the interpretation of research findings remains the surest way to establish and maintain investigatory standards.